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(g) Movement vector detecting apparatus. 

(57) A movement vector detecting apparatus com- 
prising : a movement vector detecting circuit 
for dividing an input image into a plurality of 
. blocks and for detecting a movement vector 
every block; and a movement vector calculat- 
ing circuit for extracting areas which perform 
the same movement from the movement vectors 
from the movement vector detecting circuit and 
for calculating movement vector values in such 
areas by executing a weight averaging process 
according to position information which oc- 
cupies in the area of each of the blocks. A video 
camera having such a movement vector detect- 
ing apparatus is provided. 
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BACKGROUND OF THE INVENTION 



Field of the Invention 



image vibration correcting apparatus remarkably de- 
teriorates due to the erroneous movement vectors 
which are detected from such a block. 



The present invention relates to the improvement 
of a movement vector detecting apparatus which is 
provided for an encoding apparatus or an image vibra- 
tion correcting apparatus of a moving image signal. 

Related Background Art 

Hitherto, as a movement vector detecting meth- 
od which is necessary for an encoding apparatus or 
an image vibration correcting apparatus of an image, 
there has been known a time space gradient method, 
a correlation method based on the correlation arith- 
metic operation, or a block matching method as dis- 
closed in U.S.P. No. 3,890,462, JP-B-60-46878, or 
the like. The time space gradient method is described 
in detail in B.K.P. Horn et al., "Artificial Intelligence 
17°, pages 185 to 203, 1981. The matching arithmetic 
operation is described in detail in Mono Ogami et al., 
"Information Processing", Vol. 17, No. 7, pages 634 to 
640, July, 1976. 

The time space gradient method is a method 
whereby a movement amount of an image is ex- 
pressed by d/A from a luminance difference d be- 
tween frames (or fields) and a luminance difference 
A between pixels in a picture plane. According to this 
method, the luminance difference d between frames 
(or fields) is normalized by a signal A by using a na- 
ture such that a signal which is obtained from a cam- 
era corresponds to a time average of a field period 
and, as a movement amount of the image is large, an 
edge becomes dull and the luminance difference A 
between pixels decreases. 

On the other hand, according to the block match- 
ing method (also called a template matching method), 
an input image signal is divided into blocks of a proper 
size (for instance, 8 pixels x 8 lines), a difference be- 
tween the present pixel and the pixel in a predeter- 
mined range of the preceding frame (or field) is cal- 
culated on a block unit basis, and the block of the pre- 
ceding frame (or field) in which the sum of the abso- 
lute value of such a difference becomes minimum is 
searched. A relative deviation of the relevent block in- 
dicates a movement vector of such a block. 

According to the above conventional methods, 
however, when there are a plurality of different move- 
ments in the picture plane, only movement vector of 
a low reliability can be detected from the block which 
lies over the boundary of such different movements 
because of a mismatching of the corresponding point 
due to the appearance or hiding of an object to be 
photographed or an influence by the different move- 
ments of the other objects to be photographed. There 
is a problem such that an encoding precision of the 
encoding apparatus or a correction precision of the 



5 SUMMARY OF THE INVENTION 

It is the first object of the present invention to pro- 
vide a movement vector detecting apparatus which 
can improve a detection precision of a movement vec- 

10 tor in consideration of the above problems. 

To accomplish the above object, according to a 
preferred embodiment of the present invention, there 
is disclosed a movement vector detecting apparatus 
comprising: movement vector detecting means for 

15 dividing an input image into a plurality of blocks and 
for detecting a movement vector every block; and 
movement vector calculating means for extracting 
areas in which the same motion is performed from the 
movement vectors from the movement vector detect- 

20 ing means and for calculating movement vector val- 
ues in such areas by performing a weight averaging 
process according to position information which occu- 
pies in the area of each of the blocks or movement 
vector calculating means for extracting areas in which 

25 the same motion is performed from the movement 
vectors from the movement vector detecting means 
and for calculating movement vector values in such 
areas by executing.a weight averaging process ac- 
cording to position information which occupies in the 

30 input image of each of the blocks. 

The second object of the invention is to provide 
a video camera with a vibrationproof apparatus which 
has the above movement vector detection apparatus 
and can detect and correct a vibration at a high pre- 

35 cision. 

According to a preferred embodiment of the pres- 
ent invention, there is disclosed a video camera in 
which movement vector calculating means is con- 
structed so as to calculate a movement vector value 

40 by performing a weighting process according to the 
position information which occupies in areas which 
perform the same motion of each block or position in- 
formation which occupies in an input image of each 
block, namely, according to a spatial or time- 

45 dependent adjacency degree of each block, thereby 
enabling a high accurate movement vector detection 
to be performed. 

The third object of the invention is to provide a vi- 
brationproof apparatus which detects a vibration by 

so detecting a movement vector of an image, wherein by 
always optimizing a vibrationproof area to execute the 
vibrationproof, the vibrationproof operation can be 
performed at a high precision and a high reliability. 
To accomplish the above objects, according to a 

55 preferred embodiment of the invention, there is dis- 
closed a vibrationproof apparatus comprising: move- 
ment vector detecting means for detecting a move- 
ment vector every area which is obtained by dividing 
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an image into a plurality of areas; vibration amount 
detecting means for detecting a vibration amount on 
the basis of movement vector information of an area 
corresponding to an indicated vibrationproof area 
which is obtained from the movement vector detect- s 
ing means; correcting means for correcting a vibra- 
tion; and driving means for correcting the vibration by 
driving the correcting means in accordance with the 
information from the vibration amount detecting 
means, wherein the apparatus further comprises 10 
boundary detecting means for detecting a boundary 
in the image and vibrationproof area deciding means 
for deciding the area to be subjected to a vibration- 
proof on the basis of the boundary information from 
the boundary detecting means and the information 1$ 
from the movement vector detecting means and for in- 
struction the vibrationproof area to the vibration 
amount detecting means. 

According to another preferred embodiment of 
the invention, there is disclosed a vibrationproof ap- 20 
paratus comprising: movement vector detecting 
means for detecting a movement vector every area 
which is obtained by dividing an image into a plurality 
of areas; vibration amount detecting means for.de- 
tecting a vibration amount on the basis of movement 25 
vector information of an area corresponding to an in- 
dicated vibrationproof area which is obtained from the 
movement vector detecting means; correcting means 
for correcting a vibration; and driving means for cor- 
recting the vibration by driving the correcting means 30 
in accordance with the information from the vibration 
amount detecting means, wherein the apparatus fur- 
ther comprises vibrationproof area deciding means 
for applying different weights to the information from 
the movement vector detecting means, for adding the 35 
weighted information for a predetermined time, for 
deciding the area to be subjected to the vibrationproof 
on the basis of each addition value, and for instructing 
the vibrationproof area to the vibration amount detect- 
ing means. 40 

Further another object of the invention is to pro- 
vide a video camera having the above vibrationproof 
apparatus. 

The above and other objects and features of the 
present invention will become apparent from the fol- 45 
lowing detailed description and the appended claims 
with reference to the accompanying drawings. 

BRIEF DESCRIPTION OF THE DRAWINGS 

50 

Fig. 1 is a block diagram showing a construction 
of a video camera having therein an image vibra- 
tion correcting apparatus to which the first em- 
bodiment of the present invention is applied; 
Figs. 2A to 2C are diagrams showing a flow from 55 
the input of an image signal to the judgment of a 
vibrationproof area; 

Fig. 3 is a diagram showing a binary map indica- 



tive of a vibrationproof area in the first embodi- 
ment of the invention; 

Fig. 4 is a diagram showing an example of a con- 
volution mask which is put on the vibrationproof 
area in Fig. 3; 

Fig. 5 is a diagram showing another example of 
a convolution mask which is put on the vibration- 
proof area in Fig. 3; 

Fig. 6 is a weight coefficient distribution diagram 
showing the reliability in the vibrationproof area 
in the first embodiment of the invention; 
Fig. 7 is a weight coefficient distribution diagram 
when a whole picture plane becomes a vibration- 
proof area according to the second embodiment 
of the present invention; 
Fig. 8 is a diagram showing a reliability evaluation 
function according to the total number of weight 
coefficients in the second embodiment of the in- 
vention; 

Fig. 9 is a diagram showing an example of a con- 
volution mask which is put on a vibrationproof 
area according to the third embodiment of the in- 
vention; 

Fig. 1 0 is a weight coefficient distribution diagram 
when the mask of Fig. 9 has been convoluted to 
the vibrationproof area map of Fig. 3; 
Fig. 11 is a block diagram of a video camera hav- 
ing therein an apparatus of the fourth embodi- 
ment according to the present invention; 
Fig. 12 is a diagram showing an example of a field 
image which is time-sequentially continuous in a 
tracking photographing mode; 
Fig. 13 is an optical flow which is obtained as a 
result of an arithmetic operation of two picture 
planes which are time-sequentially continuous; 
Fig. 14 is a diagram showing the result in the case 
where the optical flow of Fig. 13 has been added 
for a predetermined time; 
Fig. 15 is a block diagram of a video camera hav- 
ing therein an apparatus of the fifth embodiment 
according to the present invention; 
Fig. 16 is a model diagram of an edge existing in 
an image; 

Fig. 17 is a diagram showing a waveform when 
the edge of Fig. 16 has been subjected to a first 
order differentiation; 

Fig. 18 is a diagram showing a waveform when 
the edge of Fig. 16 has been subjected to a sec- 
ond order differentiation; 
Fig. 19 is a block diagram of a video camera hav- 
ing therein an apparatus of the sixth embodiment 
according to the present invention; 
Fig. 20 is a diagram showing a magnitude of a re- 
maining component when a vibration state has 
been modeled by a cosine curve and added for a 
predetermined time; and 
Fig. 21 is a diagram showing a magnitude of a re- 
maining component when a vibration state has 



3 



5 



EP 0 560 610 A2 



6 



been modeled by a sine curve and added for a 
predetermined time. 

DETAILED DESCRIPTION OF THE PREFERRED 
EMBODIMENTS 

The present invention will be described in detail 
hereinbelow on the basis of embodiments with refer- 
ence to the drawings. 

Fig. 1 is a block diagram showing a construction 
of a video camera having therein an image vibration 
correcting apparatus to which the first embodiment 
according to the invention has been applied. 

In Fig. 1 t reference numeral 10 denotes an object 
to be photographed; 20 an image pickup lens; 30 an 
image pickup element comprising, for example, a two- 
dimensional CCD; 40 a sample and hold (S/H) circuit 
to read out an output signal from t he image pickup ele- 
ment 30 such as a two-dimensional CCD; 50 an au- 
tomatic gain control (AGC) circuit to keep a signal lev- 
el constant; 60 an analog/digital (A/D) converter; 70 
a delay circuit to delay a color difference line- 
sequential signal from the image pickup element 30 
such as a two-dimensional CCD by only a period of 
time of two horizontal scans; 80 a color signal proc- 
essing circuit; 90 a digital/analog (D/A) converter to 
convert a color signal into a digital signal; and 95 a 
color signal output terminal. 

Reference numeral 100 denotes a low pass filter 
(LPF) to eliminate a color signal component which is 
mixed into a luminance signal; 110 an enhancer to en- 
hance a high frequency component in the luminance 
signal; 120agamma corrector; 130 a movement vec- 
tor detection circuit; 140 a microcomputer to judge the 
reliability of each block which is obtained by dividing 
a picture plane to detect a movement vector into a 
plurality of blocks; 1 50 a memory read-out control cir- 
cuit; 160 a memory to store an image of a field or a 
frame; 170 a D/A converter to convert the luminance 
signal into the digital signal; and 1 80 a luminance sig- 
nal output terminal. 

The operation of the video camera with the above 
construction will now be described. 

The object 10 is projected onto the image pickup 
element 30 by the image pickup lens 20 and is pho- 
toelectrical^ converted into an electric signal by the 
image pickup element. The S/H circuit 40 samples 
and reads out an output signal of the image pickup 
element 30. The AGC circuit 50 automatically controls 
a gain of the output signal of the S/H circuit. The A/D 
converter 60 A/D converts an output signal of the 
AGC circuit 50. The delay circuit 70 of the two-hori- 
zontal scan period of time separates the color differ- 
ence line-sequential signal from the image pickup ele- 
ment 30 into a "1 H° delay signal and a °0H + 2H° delay 
signal and supplies to a luminance signal processing 
section (circuit after the LPF 100) and the color signal 
processing circuit 80. 



The color signal processing circuit 80 produces a 
color signal. The color signal is converted into the 
analog signal by the D/A converter 90 and is output- 
ted from the color signal output terminal 95. 

5 On the other hand, the signal sent to the lumi- 
nance signal processing section is first supplied to 
the LPF 100. The LPF 100 eliminates a carrier com- 
ponent from the color difference sequential signal 
and separates the luminance signal. The enhancer 

10 110 executes a process to enhance a high frequency 
component such as an edge of the object or the like 
in order to improve a picture quality. Ordinarily, a sec- 
ondary differentiation of a video signal is added to an 
original signal. The gamma corrector 120 prevents 

15 the saturation in a highlight portion and widens.a dy- 
namic range. The movement vector detection circuit 
1 30 can be constructed by a circuit based on the fore- 
going time space gradient method or a circuit based 
on the foregoing matching arithmetic operation. In the 

20 embodiment, however, it is necessary to use a detect- 
ing method which can execute processes in a real- 
time manner. The memory 1 60 is a delay circuit to de- 
lay the luminance signal by a predetermined time (in 
the embodiment, one field time). The memory 160 

25 stores the luminance signal which is preceding by one 
: field and enables an arithmetic operation based on 
the time space gradient method or a correlation arith- 
metic operation with the present field to be per- 
formed. The microcomputer 140 first extracts the 

30 block which is a target for correction and generates a 
movement vector due to a vibration from the distribu- 
tion (optical flow) of the movement vectors which are 
detected by the movement vector detection circuit 
130. The microcomputer 140, further, executes a 

35 weighting process according to a degree of surround- 
ing state of the relevant block to the outputsignal from 
such a block and calculates a movement amount of 
the image at the moment. The memory read-out con- 
trol circuit 150 controls the read-out position of the 

40 memory 160 so as to set off (or correct) the final 
movement vector calculated by the microcomputer 
140. The luminance signal read out from the memory 
160 is converted into the analog signal by the D/A 
converter 170 and is outputted from the luminance 

45 signal output terminal 1 80. 

Amethod of evaluating the reliability of the move- 
ment vector which is executed in the microcomputer 
140 will now be described with reference to Figs. 2 to 
6. 

so Figs. 2A to 2C are diagrams showing a flow from 
the input of the image signal to the judgment of an 
area (hereinafter, referred to as a vibration proof iarea) 
to be subjected to a vibrationproof at that moment. 
Fig. 2A shows an input image which is divided into 

55 (8x6) blocks. Fig. 2B shows the result of the detec- 
tion of the.movement vector performed every block. 
A distribution diagram of such movement vectors is 
generally called an optical flow diagram. In Fig. 2B, it 
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will be undersood that areas which largely perform 
two different motions exists in the central portion and 
a peripheral portion of the picture plane. Fig. 2C 
shows the result in which the vibrationproof area has 
been extracted from the optical flow diagram. 5 

As a method of extracting the vibrationproof area 
from the movement vectors, there is a time direction 
adding method of movement vectors of each block 
disclosed in Japanese Patent Application No. 63- 
269554 which has already been filed or the like. 10 

Fig. 3 shows a binary vibrationproof area map (ef- 
fective area) in which "1 value" is allocated to the ex- 
tracted areas and "0 value 0 is allocated to the area out 
of the target 

Generally, the vibrationproof area map is multi- 15 
plied to the detected movement vectors and the ef- 
fective area is extracted and the movement vectors in 
the effective area are simple averaged by a uniform 
weight, thereby obtaining a movement amount of the 
image. 20 

Fig. 4 shows an example of a convolution mask 
which is put on the vibrationproof area map as shown 
in Fig. 3. 

The mask of Fig. 4 denotes that in the case where 
blocks adjacent to the present target blocks in the 25 
vertical and lateral directions exist, point 1 is given to 
the present target blocks, respectively. When the 
blocks adjacent to the present target blocks in the ob- 
lique directions exist, point 0.5 is given to the present 
target blocks. Due to this, the point is raised as the 30 
number of other blocks which surround the present 
target blocks is large. The reason why the point in the 
oblique direction is low is based on characteristics 
such that the image signal generally has a small 
amount of oblique component. The convolution is 35 
executed for only the effective blocks (blocks in which 
the value "1" has been inserted) as a target. Or, the 
vibrationproof area map is subjected to the AND ar- 
ithmetic operation to the result in which the convolu- 
tion has been performed to the whole picture plane, 40 
and the value of the non-vibrationproof area (invalid 
area) is held to "0". 

Fig. 5 shows an example in case of expanding the 
convolution mask from B 3 x 3" shown in Fig. 4 to "5 x 
5". 45 

A size of convolution mask or weight coefficients 
in the mask (namely, a method of giving the point) can 
be also switched in accordance with image pickup 
conditions. 

Fig. 6 shows the result in the case where the so 
mask of Fig. 4 has been convoluted to the effective 
area of the vibrationproof area map of Fig. 3. 

Coefficient values shown in Fig. 6 show the reli- 
abilities of the movement vectors which are detected 
from the relevant block. It is sufficient to execute an 55 
averaging process of the movement vectors in accor- 
dance with the weights. 

The read-out position of the image from the mem- 



ory is decided by using the movement vectors ob- 
tained by the above processes and the image in 
which the unpleasant vibration component has been 
corrected is outputted from a luminance signal output 
terminal 180. 

(Second embodiment) 

According to the above first embodiment, the 
weight averaging process of the movement vectors 
has been executed to the effective area of the vibra- 
tionproof area map in accordance with the result in 
which a certain mask has been convoluted. In the 
second embodiment, however, the reliability of the 
movement vector itself obtained in the first embodi- 
ment is newly evaluated by using the total number of 
weight coefficients obtained as a result of the convo- 
lution of a certain mask, namely, by using the total 
number of points. The second embodiment can be 
also used together with the first embodiment. 

Fig. 7 shows the result in which the convolution 
mask of Fig. 4 was put in the case where it is deter- 
mined that all of the "8x6° blocks in the picture plane 
are effective. 

In this case, the total number of all weight coef- 
ficients in Fig. 7 is equal to 282 and is the upper limit 
of the such a total number. Since the reliability of the 
movement vectors which are detected from the rele- 
vant area is high as the area of the vibrationproof area 
is large, the reliability of the movement vector can be 
newly evaluated from the magnitude of the total num- 
ber of weight coefficients. 

Fig. 8 shows a reliability evaluation function ac- 
cording to the total number of weight coefficients. 

When the total number of points is small, the ex- 
tracted area is small orthe extracted areas separately 
exist. In any of the above two cases, since a possibil- 
ity such that the movement vectors were erroneously 
detected is high, a small coefficient less than T is 
multiplied, thereby suppressing an influence on the 
subsequent processes. Now, assuming that the num- 
ber of blocks in the picture plane is equal to n M x N°, 
the distribution of the points is expressed as follows. 
0 ^ (the total number of points) ^ 7MN - 4M - 4N 
+ 2 

The above processes are executed by the micro- 
computer 140 in Fig. 1. 

(Third embodiment) 

In the above first and second embodiments, the 
reliability has been evaluated by using the informa- 
tion of the spatial arrangement of the target blocks as 
a clue. According to the third embodiment, however, 
such a concept is expanded in a time base direction. 

Generally, a correlation in the time direction of a 
moving image is high. For instance, when the object 
is traced and photographed, the position of the object 
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which occupies in the picture plane is strongly restrict- 
ed at the position locating in the preceding picture 
plane (preceding field) and a possibility such that the 
object appears every field without a correlation is ex- 
tremely small. Namely, a possibility such that the ef- 5 
fective area in the present picture plane appears at a 
position near the area which has been regarded as an 
effective area in the preceding picture plane is high. 
Therefore, the area which has been judged as an ef- 
fective area in the present picture plane is set to the 10 
peak center and a map having a gentle weight coef- 
ficient distribution is formed and the AND is got be- 
tween such a map and the vibrationproof area map 
which will be obtained in the next picture plane. Thus, 
it is possible to evaluate the reliability utilizing the cor- 15 
relation in the time direction of the positions in the pic- 
ture plane of a certain area. 

An embodying procedure will now be described 
with reference to Figs. 9 and 10. 

Fig. 9 shows a convolution mask which is put on 20 
the vibrationproof area map obtained in the present 
picture plane. Since it is considered that a deviation 
of the position of an object between images occurs at 
the same probability in every direction of 360°, the 
weight coefficient distribution of the mask is also iso- 25 
tropic. 

Fig. 1 0 shows the result in which the mask shown 
in Fig. 9 has been convoluted to the vibrationproof 
area map shown in Fig. 3. 

By executing an AND arithmetic operation be- 30 
tween the weight coefficient distribution shown in Fig. 
10 and the vibrationproof area map which will be ob- 
tained in the next picture plane, the reliability of each 
block can be easily evaluated. 

The above processes are executed by the micro- 35 
computer 140 in Fig. 1. 

According to the foregoing embodiment, the reli- 
ability of the movement vector which is detected is 
evaluated by the spatial or time-dependent adjacency 
degree (approach degree) of each block and the 40 
weighting process according to the reliability evalua- 
tion is executed to the movement vector. Therefore, 
it is possible to improve the precision of the moving 
image encoding apparatus or image vibration correct- 
ing apparatus whose precision has remarkably been 45 
deteriorated by the movement vectors of low reliabil- 
ities which are detected from the block including a 
boundary portion in which different motions inherent 
exist. 

According to the above embodiment as descri- so 
bed above, the apparatus comprises: the movement 
vector detecting means for dividing an input image 
into a plurality of blocks and for detecting movement 
vectors every block; and the movement vector calcu- 
lating means for extracting the areas in which the 55 
same motion is performed from the movement vec- 
tors from the movement vector detecting means and 
for calculating the movement vector values in such 



areas by executing the weight averaging process ac- 
cording to the position information which occupies in 
the areas of each of the blocks or the movement vec- 
tor calculating means for extracting the areas in which 
the same motion is performed from.the movement 
vectors from the movement vector detecting means 
and for calculating the movement vector values in 
such areas by executing the weight averaging proc- 
ess according to the position information which occu- 
pies in the input image of each block, wherein the 
movement vector value is calculated by executing the 
weighting process according to the position informa- 
tion which occupies in the areas in which the same 
motion is performed of each block or the position in- 
formation which occupies in the input image of each 
block, namely, according to the spatial or time- 
dependent adjacency degree of each block. 

Consequently, the detection precision of the 
movement vector can be improved. 

The fourth to sixth embodiments of the present 
invention will now be described. 

Those embodiments are based as a prerequisite 
on the vibrationproof apparatus in which movement 
vector information of the area corresponding to the 
designated vibrationproof area on the picture plane is 
detected and the vibration amount is detected on the 
basis of the movement vector information and the vi- 
bration correcting means is made operative to there- 
by correct the vibration. The deciding process of the 
vibrationproof area to detect the vibration is opti- 
mized. 

Prior to explaining those embodiments, the back- 
ground of such a kind of vibrationproof apparatus will 
be first described. 

In a photographing optical apparatus such as 
camera, video camera, electronic camera, or the like, 
the camera vibration makes it hard to see the image 
and becomes a cause of every malfunction irrespec- 
tive of a measuring instrument for an industrial use or 
an apparatus for a public use. Particularly, in the pho- 
tographing during the walk, the photographing from a 
moving vehicle, or the photographing at a location 
where there are much vibrations, the vibration of the 
picture plane easily occurs. Therefore, various kinds 
of methods of correcting the vibration of the picture 
plane have conventionally been proposed. 

For instance, according to the vibrationproof 
camera disclosed in JP-A-61-248681 , an incident im- 
age is converted into an electric signal by an image 
pickup system comprising a lens system and a pho- 
toelectric converting element. The eiectric signal is 
further subjected to predetermined signal processes 
by the circuit at the next stage and is converted into 
the television image signal and supplied to a monitor 
apparatus. The TV image signal is also supplied to an 
image vibration detecting circuit. By getting a corre- 
lation between two picture planes which is away from 
each other by a predetermined. time interval, the 
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magnitude and direction of the image vibration are de- 
tected. On the basis of the result of the detection, the 
lens system is driven and controlled in such a direc- 
tion as to set off the image vibration. 

On the other hand, according to "Picture Plane 5 
Vibration Correcting Apparatus 0 , The Japan Society 
of Television, Technical Report Vol. 11, No. 3, pages 
43 to 48, PPOE, 87 - 1 2, May, 1 987, between two pic- 
ture planes which are away from each other by a pre- 
determined time interval, the correlation is obtained 10 
every small area (hereinafter, referred to as block) 
which is obtained by equivalent^ dividing the picture 
plane into 140 small areas, thereby detecting the 
magnitude and direction of the image vibration with 
respect to the whole picture plane. The results of the 15 
detection are subjected to a decision by majority, 
namely, the area of the maximum area in which the 
detected movement vectors are preferably arranged 
is set to a targetforthe vibrationproof, thereby judging 
the magnitude and direction of the picture plane vi- 20 
bration. The area corresponding to the preceding pic- 
ture plane is extracted from the memory in which the 
input image has been stored so as to set off the pic- 
ture plane vibration. Enlarging and interpolating proc- 
esses are executed and the resultant data is output- 25 
ted as an output image. 

Further, according to Tracking Area Deciding 
Method" which has already been filed by the same 
applicant as the present invention in JP-A-2-11 7276, 
a tracking area is automatically judged by accumulat- 30 
ing the movement vectors detected every block for a 
predetermined time, and the optical correcting sys- 
tem (variable vertical angle prism) put in front of the 
lens system is driven so as to correct the movement 
vectors detected in such an area, thereby suppress- 35 
ing the picture plane vibration. 

The vibrationproof denotes a function to correct 
the camera vibration. The tracking denotes a function 
for the camera to trace a moving object The principle 
processes and control methods of those functions are 40 
equal. 

In the above example, however, there is a prob- 
lem such that an outline of the area to be traced is ob- 
scure because of a deterioration in precision of the 
movement vectors in the boundary area of the area 45 
to be traced and the precision of the movement vec- 
tors indicative of the picture plane vibration is deterio- 
rated by the movement vectors of low reliabilities 
which are generated from the block including the out- 
line portion. Even in the case other than the tracking 50 
image pickup operation, for instance, when the oper- 
ator wants to perform the vibrationproof to a back- 
ground existing in a peripheral portion in the picture 
plane, when some moving object or motion object ex- 
ists in the central portion, the above problem occurs 55 
in the boundary region of such an object. The causes 
of such a problem are based on the hiding or appear- 
ance of the image in the outline portion of the object 



to be photographed in association with the movement 
or motion of the object. In such a case, since there is 
no guarantee of the correspondence between the 
front and after images, only the movement vectors of 
low reliabilities can be obtained. 

As another problem, according to the vibration- 
proof camera disclosed in JP-A-61-248681 as a first 
example mentioned above, the magnitude and direc- 
tion of the image vibration are detected by obtaining 
a correlation between the images which are away 
from each other by a predetermined time. Therefore, 
it is desirable to preserve one picture in one image. It 
is difficult to apply such a technique to the case where 
a partial motion or modification occurs in the image 
or the case where the background continuously 
changes in the tracking photographing mode. There- 
fore, such a technique can be used in only a limited 
application field. 

According to "Picture Plane Vibration Correcting 
Apparatus" as a second example mentioned above, 
on the other hand, since the area (hereinafter, refer- 
red to as a vibrationproof area) to be at present sub- 
jected to the vibrationproof is determined by a meth- 
od of decision by majority, in a still image photograph- 
ing mode such t hat the operator wants to trace a small 
object or to perform the vibrationproof to the back- 
ground, in the case where the object to be photo- 
graphed is larger than the background portion and 
the movement vectors of the object are preferably ar- 
ranged, or the like, the area against the purpose of 
the photographer is subjected to the vibrationproof. 

According to "Tracking Area Deciding Method" as 
a third example mentioned above, the picture plane 
is divided into a plurality of areas (blocks) and the 
movement vectors detected every block are accumu- 
lated for a predetermined time, thereby deciding the 
vibrationproof area. Therefore, such a method is 
strong even for a partial motion in the image and can 
realize the vibrationproof at the highest fidelity ac- 
cording to the will of the photographer. There is, how- 
ever, a ease where the accumulation value which is 
obtained by accumulating the movement vectors de- 
tected from the vibrationproof area for a predeter- 
mined time is not so enough small. Such a phenom- 
enon occurs, particularly, in the case where the per- 
iod of time is longer than the accumulation time and 
the vibration of a low frequency has occurred. In such 
a case, the division of the vibrationproof area/non- 
vibrationproof area cannot be correctly performed. 

it is the first object of the embodiment is to pro- 
vide a vibrationproof apparatus which can accurately 
perform the vibrationproof of an object to be traced 
and, further, the vibrationproof of the background 
without being influenced by the moving object or mo- 
tion object 

The second object of the embodiment is to pro- 
vide a vibrationproof apparatus which can accurately 
execute the vibrationproof even under any photo- 
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graphing situation. 

According to the embodiment, therefore, the ap- 
paratus comprises: boundary detecting means for de- 
tecting a boundary in an image; and vibrationproof 
area deciding means for deciding an area to be sub- 5 
jected to the vibrationproof on the basis of boundary 
information from the boundary detecting means and 
information from movement vector detecting means 
and for instructing the vibrationproof area to vibration 
amount detecting means, wherein the boundary in a 10 
picture plane is detected by a movement vector dis- 
tribution or a Laplacian arithmetic operation of an in- 
put image and the area to be subjected to the vibra- 
tionproof is determined by the detected boundary and 
the movement vector distribution. 15 

On the other hand, the apparatus has vibration- 
proof area deciding means for applying different 
weights to the information from the movement vector 
detecting means, for adding the weighted information 
for a predetermined time, for deciding the area to be 20 
subjected to the vibrationproof on the basis of the ad- 
dition values, and for instructing the vibrationproof 
area to the vibration amount detecting means. The vi- 
bration area is determined from the magnitudes of the 
addition values which are obtained by applying differ- 25 
ent weights to the movement vector information. 

(Fourth embodiment) 

Fig. 11 is a block diagram showing a video cam- 30 
era having therein a vibrationproof apparatus accord- 
ing to the fourth embodiment of the present invention, 
in the diagram, reference numeral 110 denotes an ob- 
ject to be photographed; 111 a variable vertical angle 
prism which can change the optical axis direction; 112 35 
an image pickup lens; 113 an image pickup element 
comprising, for example, a two-dimensional CCD; 
and 114 a signal processing circuit to execute a gam- 
ma correction, a blanking process, an adding process 
of a sync signal, and the like to an output image signal 40 
from the image pickup element 113. A television sig- 
nal of, for instance, the NTSC standard is outputted 
from an output terminal 115. Y denotes a luminance 
signal. H. SYNC indicates a horizontal sync signal. V. 
SYNC denotes a vertical sync signal. Reference nu- 45 
meral 116 denotes a delay circuit to delay the lumi- 
nance signal Y by a predetermined time. For example, 
the delay circuit 11 6 is constructed by a field memory 
of the FIFO (First-in First-out) type. Reference nu- 
meral 117 denotes a block division pulse generation 50 
circuit to generates gate pulse to divide an image sig- 
nal which is being scanned into predetermined blocks 
set on a picture plane. 

Reference numerals 118 and 119 denote division 
circuits each for dividing the luminance signal Y in ac- 55 
cordance with the output pulses from the block divi- 
sion pulse generation circuit 117. Each of the division 
circuits 118 and 119 generates the input luminance 



signal Y in a lump every block set on the picture plane. 
Practically speaking, each division circuit comprises 
a gate circuit section whose opening and closing op- 
erations are controlled by the output pulse of the gen- 
eration circuit 117 and a memory section to store a 
passage signal of the gate circuit section. 

Reference numeral 120 denotes a movement 
vector detection circuit for comparing the signal of the 
present picture plane and the signal of the picture 
plane of a predetermined time outputted from the de- 
lay circuit 116 and for obtaining the movement vector 
every divided block. Reference numeral 121 denotes 
a memory to store the movement vector information 
in each section on the picture plane and 122 indicates 
a Laplacian operation circuit to calculate a Laplacian 
of the movement vector. 

Reference numeral 123 denotes a vibrationproof 
area decision circuit to decide the area to be subject- 
ed to the vibrationproof by using the Laplacian of the 
movement vector which has been calculated in the 
Laplacian operation circuit 122 as a due; 124 a vibra- 
tion amount detection circuit to detect a vibration 
amount from the luminance signal Y. For instance, the 
vibration amount detection circuit 124 is constructed 
by a correlation arithmetic operation circuit to per- 
form a representative point matching or the like or an 
arithmetic operation circuit of the time space gradient 
method. Reference numeral 125 denotes a driving 
circuit to drive an actuator 126, which will be ex- 
plained hereinlater, in accordance with an output of 
the vibration amount detection circuit 124. Reference 
numeral 126 denotes the actuator to change a degree 
of vertical state for the optical axis of the. variable 
vertical angle prism 111. By changing the vertical an- 
gle of the prism 111, the angle of deviation of the out- 
going optical axis to the incident optical axis of the 
prism 111 can be controlled. 

In the above construction, the object image 
which passed through the prism 111 and lens 112 en- 
ters the image pickup element 113, by which it is pho- 
toelectrical^ converted into the electric image signal. 
The image signal is supplied to the signal processing 
circuit 114 and is subjected to the gamma correction, 
blanking process, adding process of the sync signal, 
and the like. The luminance signal Y which is output- 
ted from the signal processing circuit 114 is directly 
supplied to the division circuit 119. The luminance sig- 
nal Y is also delayed by the delay circuit 116 by the 
time of one field period (about 16.7 msec) and is sent 
to the division circuit 118. Thus, the division circuits 
118 and 119 divide the preceding picture plane into °M 
x N" blocks in accordance with the output pulses of 
the block division pulse generation circuit 117. The 
movement vector detection circuit 120 can be con- 
structed by a circuit based on the time space gradient 
method or a circuit based on the matching arithmetic 
operation. However, it is necessary to use a "circuit 
construction which can execute the processes in a 
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real-time manner in the embodiment. 

The time space gradient method has been descri- 
bed in B.K.P. Horn et al., "Artificial Intelligence 17", 
pages 185 to 203, 1981. The matching arithmetic op- 
eration has been described in Morio Ogami et al., "In- 5 
formation Processing", Vol. 17, No. 17, pages 634 to 
640, July, 1976. 

The operations from the movement vector detec- 
tion circuit 120 to the vibrationproof area decision cir- 
cuit 123 will now be practically explained with refer- w 
ence to Figs. 12 to 14. A distribution of the movement 
vectors indicative of a speed field of a whole picture 
plane is called an optical flow. 

Fig. 12 shows an example of field picture planes 
which are time-sequentially continuous when they 15 
have been photographed in the tracking mode. Fig. 
13 shows an example of an optical flow which is ob- 
tained as a result of the arithmetic operation between 
the two picture planes which are time-sequentially 
continuous. A movement vector indicative of the vi- 20 
bration occurring during the tracking photographing 
operation can be observed from a region (1) in the 
central portion of the picture plane where an object 
(car) exists. On the other hand, a movement vector in 
a background portion (2) is obtained by synthesizing 25 
the movement vector indicative of a flow of the image 
in association with the tracking photographing oper- 
ation and the movement vector due to the vibration. 

From only Fig. 1 3, it is impossible to correctly de- 
cide which one of the regions is set to the vibration- 30 
proof area. The movement vector in a lower edge por- 
tion (3) in Fig. 13 is a vector which has been outputted 
from a region where enough image informrtion 
doesn't exist in the input image and the reliability is 
extremely low and such a vector can be regarded as 35 
a noise. When the present invention is put into prac- 
tical use, the area in which image information lacks 
needs to be previously eliminated. 

Fig. 14 is a diagram showing an example of the 
result in which the optical flow as shown in Fig. 13 has AO 
been added for a predetermined time (1 .3 seconds in 
this example). 

In Fig. 14, a region of an object (car) of a small ad- 
dition value can be observed in the central portion of 
the picture plane. In this example, the photographer 45 
tries to arrange an object to the central portion of the 
picture plane. By adding the optical flow for a prede- 
termined time, a target to be photographed now by 
the photographer can be judged. 

This is based on an idea such that "Since the 50 
photographer tries to stop the imate in the vibration- 
proof area, the image in the vibrationproof area 
swings at random around the position at which the 
photographer tries to stop the image as a center". 

The addition result of the movement vectors de- 55 
tected in the vibrationproof area is converged to a 
small value because of the random characteristics of 
the movement vectors. On the other hand, such a- 



.state doesn't occur in the non-vibrationproof area, so 
that the division. between the vibrationproof area and 
the non-vibrationproof area can be performed from 
the magnitude of the addition value of the movement 
vectors. 

The movement vectors detected by the move- 
ment vector detection circuit 120 are accumulated 
into the memory 121 for a predetermined time. After 
that, the accumulation value is supplied to the Lapla- 
cian operation circuit 122. In the Laplacian operation 
circuit 122, the Laplacian of the movement vectors 
detected by the movement vector detection circuit 
120 is calculated, thereby obtaining a block of a large 
change in movement vector. The calculation of the 
Laplacian is defined by the following equations. 

V*u = Pu/dx 2 + d*u/dy 2 

V*v = Pvldx 2 + d 2 vldy 2 
where, u and v denote speeds in the x direction and 
y direction. 

Since the area in which the result of the Lapla- 
cian arithmetic operation is large corresponds to a 
portion in which the movement vector suddenly 
changes, the block such that the accurate movement 
vector cannot be detected can be specified in princi- 
ple from the magnitude of the result of the Laplacian 
operation. 

In the vibrationproof area decision circuit 123, the 
vibrationproof area to which the photographer wants 
to perform the vibrationproof at present is accurately 
decided from the distribution of the result of the Lap- 
lacian arithmetic.operation. The vibrationproof area is 
first coarsely determined from the distribution of the 
addition values of the movement vectors which are 
detected for the whole area of the picture plane for a 
predetermined time. Further, the area in which the 
accurate movement vectors were detected is selected 
from the result of the Laplacian operation. In the vi- 
bration amount detection circuit 124, the movement 
vectors in the area decided by the vibrationproof area 
decision circuit 123 is calculated. Specifically speak- 
ing, among the movement vectors detected by the 
movement vector detection circuit 1 20, the average of 
the movement vectors which are outputted from the 
block corresponding to the vibrationproof area can be 
also obtained. Or, the movement vectors can be also 
again newly accurately detected in the vibrationproof 
area. The vibration amount detected by the vibration 
amount detection circuit 124, namely, the picture 
plane vibration amount is supplied to the driving cir- 
cuit 125. The actuator 126 is driven by the driving cir- 
cuit 125 and a proper angle of deviation is given to the 
prism 111 and the vibration amount is optically cor- 
rected. 

(Fifth embodiment) 

According to the above first embodiment, the 
Laplacian of the optical flow has been calculated to 
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decide the vibrationproof area. However, the Lapla- 
cian of the input image itself can be also calculated 
and such a case will now be described hereinbelow as 
a fifth embodiment. 

Fig. 1 5 is a block diagram of a video camera hav- 
ing therein a vibrationproof apparatus according to 
the fifth embodiment of the present invention. 

Fig. 1 5 differs from Fig. 1 1 with respect to a point 
that the image signal (luminance signal Y) from the 
signal processing circuit 114 is supplied to the mem- 
ory 121 in place of the signal from the movement vec- 
tor detection circuit 120. 

The input image is once stored in the memory 
121 and the Laplacian operation is executed to the in- 
put image by the Laplacian operation circuit 122. 

Effects of the Laplacian will now be described 
with reference to Figs. 16 to 18. 

Fig. 16 is a model diagram of an edge existing in 
the image. Fig. 17 is a diagram showing the result in 
which the edge of Fig. 1 6 has been first order differ- 
entiated. Fig. 18 is a diagram showing the result in 
which the result of Fig. 1 7 has further been first order 
differentiated. Therefore, Fig. 18 shows the result of 
the second order differentiation in Fig. 16, that is, the 
Laplacian. 

A zero intersection point shown in Fig. 18 is gen- 
erally called a zero crossing point and shows an edge 
portion of the image. The edge corresponds to the lo- 
cation at which the hiding or appearance of the image 
mentioned in the foregoing first embodiment occurs. 

In the Laplacian operation circuit 122, the Lapla- 
cian of an image is calculated. In the vibrationproof 
area decision circuit 123, the block including the zero 
crossing point is found out from the result of the out- 
put of the Laplacian operation circuit 122 and the 
movement vectors detected by the movement vector 
detection circuit 120 and is set into a non-vibration- 
proof area. The other areas are determined as a vi- 
brationproof area. In the vibration amount detection 
circuit 124, the movement vectors in the area decided 
by the vibrationproof area decision circuit 123 is cal- 
culated in a manner similar to the first embodiment. 
Specifically speaking, among the movement vectors 
detected by the movement vector detection circuit 
1 20, the average of the movement vectors which are 
outputted from the block corresponding to the vibra- 
tionproof area can be also obtained or the movement 
vectors can be also again newly accurately detected 
in the vibrationproof area. The vibration amount de- 
tected by the vibration amount detection circuit 124, 
namely, the picture plane vibration amount is supplied 
to the driving circuit 125. The actuator 126 is driven 
by the driving circuit 125. A proper angle of deviation 
is given to the prism 111 and the vibration amount is 
optically corrected. 

According to the above fourth and fifth embodi- 
ments, the boundary of the vibrationproof area that is 
presumed from the distribution of the addition values 



of the movement vectors or the distribution of the ad- 
dition values of the input image can be clarified by a 
simple arithmetic operation. Thus, the vibrationproof 
can be performed to the background portion without 

5 being influenced by the moving object or motion obr 
ject On the contrary, the vibrationproof of an object 
to be traced can be also similarly stably performed. 
The method of deciding the vibrationproof area men- 
tioned above can be widely applied to not only the TV 

10 camera for photographing but also an industrial TV 
camera, monitoring camera, or the like. There is prac- 
tically a remarkable advantage. 

(Sixth embodiment) 

15 

Fig. 19 is a block diagram of a video camera hav- 
ing therein a vibrationproof apparatus according to 
the sixth embodiment of the present invention. In Fig. 
19, the same portions as those in Fig. 11 are desig- 

20 nated by the same reference numerals. 

In Fig. 19, reference numerals 201 and 202 de- 
note memories. Proper weights are applied to the 
movement vectors in each portion on the picture 
plane detected by the movement vector detection cir- 

25 cuit 120 and the resultant weighted values are accu- 
mulated and stored into those memories. Reference 
numeral 203 denotes a comparison and integration 
processing circuit to compare and integrated various 
weight addition values of the movement vectors. Ref- 

30 erence numeral 204 indicates a vibrationproof area 
decision circuit to decide the vibrationproof area from 
an output of the comparison and integration process- 
ing circuit 1 38 and an output of the movement vector 
detection circuit 120. 

35 The operation in the above construction will now 
be described. 

The object image which passed through the 
prism 111 and the lens 112 enters the image pickup 
element 113, by which it is photoelectrical^ converted 

40 into the electric image signal. The image signal is sup- 
plied to the signal processing circuit 114 and is sub- 
jected to processes such as gamma correction, 
blanking process, adding process of a sync signal, 
and the like. The luminance signal Y which is output- 

45 ted from the signal processing circuit 114 is directly 
supplied to the division circuit 119. The luminance sig- 
nal Y is also delayed by the delay circuit 116 by the 
time of one field period (about 1 6.7 msec) and the de- 
layed signal is supplied to the division circuit 118. The 

so division circuits 118 and 119 divide the preceding pic- 
ture plane into "M x N° blocks in accordance with out- 
put pulses from the block division pulse generation 
circuit 1 1 7. The movement vector detection circuit 1 20 
can be constructed by the circuit based on the time 

55 space gradient method or the circuit based on the 
matching arithmetic operation. However, it is neces- 
sary to use a circuit construction which can execute 
the processes in a real-time manner in the embodi- 
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merit 

The operations from the movement vector detec- 
tion circuit 120 to the vibrationproof area decision cir- 
cuit 123 will be further described with reference to 
Figs. 12 to 14 again. 5 

Fig. 12 is a diagram showing an example of field 
picture planes which are time-sequentially continuous 
when they have been photographed in the tracking 
mode. Fig. 13 shows an example of an optical flow 
which is obtained as a result of the arithmetic opera- 10 
tion between the two picture planes which are time- 
sequentially continuous. The movement vectors indi- 
cative of the vibration occurring during the tracking 
photographing operation can be observed from the 
region (1) in the central portion of the picture plane 15 
where the object (car) exists. On the other hand, the 
movement vectors in the background portion (2) are 
obtained by synthesizing the movement vectors indi- 
cative of the flow of the image in association with.the 
tracking photographing operation and the movement 20 
vectors due to the vibration. 

From only Fig. 13, it is impossible to correctly de- 
cide which one of the regions is a vibrationproof area. 
The movement vectors in the lower edge portion (3) 
in Fig. 1 3 are vectors outputted from the area in which 25 
there is no enough image information in the input im- 
age and the reliabilities are extremely low, so that 
those movement vectors can be regarded as a noise. 
When the invention is put into practical use, the area 
in which image information lacks needs to be previ- 30 
ously eliminated. 

Fig. 14 is a diagram showing an example of the 
result in which the optical flow as shown in Fig. 13 
was added for a predetermined time (1.3 seconds 
here). 35 

In Fig. 14, the area of the object (car) of a small 
addition value can be observed in the central portion 
of the picture plane. In the example, the photographer 
tries to arrange the object to be photographed to the 
central portion of the picture plane. By adding the opt- 40 
ical flow for a predetermined time, it is possible to 
judge a target which the photographer tries to photo- 
graph at present 

Such a method is based on the idea such that 
"Since the photographer tries to stop the image in the 45 
vibrationproof area, the image in the vibrationproof 
area swings at random around the position at which 
the photographer tries to stop the image as a center". 

The addition result of the movement vectors de- 
tected in the vibrationproof area is converged to a 50 
small value because of the random characteristics of 
the movement vectors. On the other hand, such a 
conversion doesn't occur in the non-vibrationproof 
area. Thus, the vibrationproof area and the non-vibra- 
tionproof area can be divided from the magnitude of 55 
the addition value of the movement vectors. 

The movement vectors detected by the move- 
ment vector detection circuit 120 are accumulated 



into the memories 201 and 202 for a predetermined 
time. After that they are supplied to the comparison 
and integration processing circuit 203. 

The operations of the memories 201 and 202 will 
now be described in detail with reference to Figs. 20 
and 21. 

Fig. 20 shows the addition result when a vibration 
state has been modeled by a cosine curve of an am- 
plitude "1" and the modeled values have been added 
for a predetermined time (T), namely, a magnitude of 
the remaining component. An axis of ordinate indi- 
cates a remaining component amount which has been 
standardized for the accumulation time T. An axis of 
abscissa indicates a vibration frequency. 

In Fig. 20, a solid line (J) shows a remaining com- 
ponent when the movement vectors have been added 
by a uniform weight for a predetermined time. A brok- 
en line ® indicates a remaining component when a 
weight function which is convex in the upward direc- 
tion and whose weight is large as a time point is close 
to the present time has been multiplied to the move- 
ment vectors and the resultant movement vectors 
have been added. A short broken line (D shows a re- 
maining component when the movement vectors 
have been added by using a weight function whose 
weight increases at regular intervals as a time point 
is close to the present time. A long broken line ® 
shows a remaining component when the movement 
vectors have been added by using a weight function 
which is convex in the upward direction and whose 
weight is large as a time point is close to the present 
time. 

Fig. 21 shows a magnitude of a remaining com- 
ponent when a vibration state has been modeled by 
a sine curve of an amplitude "1 " and the modeled val- 
ues have been added for a predetermined time (T). 
The contents of Fig. 21 will be understood in a manner 
similar to Fig. 20. 

Fig. 20 shows the case where the DC component 
exists and corresponds to the panning operation or 
the operation in the tracking photographing mode. On 
the other hand, Fig. 21 shows the case where no DC 
component exists and corresponds to the operation 
in the still photographing mode (state in which the 
photographer sets the camera main body into a still 
state and tries to photograph). It will be obviously un- 
derstood from Figs. 20 and 21 that in a region of a high 
vibration frequency, there is not so large difference 
between the methods of applying weights and that, 
on the contrary, in a region of a low vibration frequen- 
cy, a large difference occurs in the appearance of the 
remaining component in dependence on the method 
of applying the weight. 

The memories 201 and 202 are memories to re- 
alize, for example, four.kinds of weight adding meth- 
ods as shown in Figs. 20 and 21 and have elated pho- 
tographing modes. That is, in those diagrams, in case 
of adding by a uniform load (weight) as shown by the 
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solid line ®, there is a large difference between the 
time addition value of the movement vectors due to 
the vibration of a low frequency and the time addition 
value of the movement vectors by the vibration of a 
high frequency, so that the area division in a photo- 5 
graphing situation in which the DC component exists 
is elated. On the contrary, in case of adding by using 
the load which becomes significant in an exponential 
function manner as the data is close to the latest data 
as shown by the long broken line ®, the time addition 1 o 
value of the movement vectors by vibrations of almost 
all of the frequencies is set to a small value and a still 
photographing situation having no DC component is 
elated. 

However, since the photographing state cannot 15 
previously been known, in the comparison and inte- 
gration processing circuit 203, the time addition val- 
ues (namely, remaining components) of the move- 
ment vectors outputted from the memories 201 and 
202 to accumulate the movement vectors with differ- 20 
ent weights are compared and integrated, thereby 
first presuming the present photographing situation. 
Subsequently, for instance, in the case where it is pre- 
sumed that the present photographing situation is a 
tracking photographing state, a more significance is 25 
paid to the weight addition result shown by the solid 
line ® in Figs. 20 and 21 and the addition result is out- 
putted. On the contrary, in the case where it is pre- 
sumed that the present state is the still photographing 
state, a more significance is paid to the weight addi- 30 
tion result shown by the long broken line ® in Figs. 

20 and 21 and the addition result is outputted. Or, the 
memory which adaptively outputs in accordance with 
the photographing situation can be also switched. In 

the case where the photographing state cannot be 35 
classified to any one of those situations, the average 
between the states shown by the solid line ® and 
long broken line ® in Figs. 20 and 21 can be also set. 
The intermediate weight addition as shown by the 
broken line @ and short broken line (D in Figs. 20 and 40 

21 can be also performed. 

In the vibrationproof area decision circuit 204, the 
vibrationproof area is accurately decided from the 
magnitude relation of the addition values of the move- 
ment vectors which are outputted from the compari- 45 
son and integration processing circuit 203. In the vi- 
bration amount detection circuit 124, the movement 
vectors in the area decided by the vibration area de- 
cision circuit 204 are arithmetically operated. Practi- 
cally speaking; among the movement vectors detect- so 
ed by the movement vector detection circuit 120, the 
average of the movement vectors which are outputted 
from the block corresponding to the vibrationproof 
area can be also used. Or, the movement vectors can 
be also again newly accurately detected in the vibra- 55 
tionproof area. The vibration amount detected by the 
vibration amount detection circuit 124, namely, the 
picture plane vibration amount is supplied to the driv- 



ing circuit 125 in a manner similar to that mentioned 
above. The actuator 1 26 is driven by the driving circuit 
125. A proper angle of deviation is added to the prism 
111 and the vibration amount is optically corrected. 

According to the sixth embodiment, by providing 
a plurality of memories to apply different weights to 
the movement vectors and to accumulate and add the 
weighted movement vectors, the vibrationproof area 
can be accurately decided in every photographing sit- 
uation. In the embodiment as well, the vibrationproof 
area deciding method can be widely applied to not 
only the photographing TV camera but also the indus- 
trial TV camera, monitoring camera, orthe like. There 
is a remarkable advantage in the practical use. 

As described above, according to each of the 
above embodiments, the apparatus comprises: the 
boundary detecting means for detecting a boundary 
in an image; and the vibrationproof area deciding 
means for deciding the area to be subjected to the vi- 
brationproof on the basis of the boundary information 
from the boundary detecting means and the informa- 
tion from the movement vector detecting means and 
for instructing the vibrationproof area to the vibration 
amount detecting means, wherein the boundary in 
the picture plane is detected by the movement vector 
distribution or the Laplacian arithmetic operation of 
the input image. The area to be subjected to the vi- 
brationproof is determined by the detected boundary 
and the movement vector distribution. 

Therefore, the vibrationproof of the object to be 
traced, further, the vibrationproof of the background 
can be accurately performed without being influ- 
enced by the moving object or motion object. 

On the other hand, the apparatus has the vibra- 
tionproof area deciding means for applying different 
weights to the information from the movement vector 
detecting means, for adding the weighted information 
for a predetermined time, for deciding the area to be 
subjected to the vibrationproof on the basis of the ad- 
dition values, and for instructing the vibrationproof 
area to the vibration amount detecting means. The vi- 
brationproof area is decided from the magnitudes of 
the addition values which are obtained by multiplying 
different weights to the movement vector information. 
Consequently, the accurate vibrationproof can be 
executed under every photographing situation. 



Claims 

1 . A movement vector detecting apparatus compris- 
ing: 

a) movement vector detecting means for div- 
iding an input image into a plurality of blocks 
and for detecting movement vectors every 
block; and 

b) movement vector calculating means for ex- 
tracting areas in which the same motion is 
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performed from the movement vectors from 
said movement vector detecting means and 
for calculating the movement vector values in 
said areas by executing a weight averaging 
process according to position information 5 
which occupies in the area of each of said 
blocks. 

2. An apparatus according to claim 1 , further having 
image pickup means for photoelectrically con- 10 
verting an object image-formed on an image 
pickup surface and for outputting an image pick- 
up signal. 

3. An apparatus according to claim 2, wherein said 15 
input image is an image formed on the image 
pickup surface of said image pickup means, and 

said movement vector detecting means detects 
the movement vectors from the image pickup sig- 
nal. 20 

4. An apparatus according to claim 3, wherein said 
movement vector detecting means obtains the 
movement vectors from a luminance signal which 

is obtained by eliminating a color signal compo- 25 
nent from the image pickup signal. 

5. An apparatus according to claim 1, wherein said 
movement vector detecting means generates a 
distribution (optical flow) of the movement vec- 30 
tors on the picture plane. 

6. An apparatus according to claim 5, wherein said 
movement vector calculating means extracts the 
blocks which output the movement vectors due to 35 
a vibration as a target for correction from said opt- 
ical flow and applies weights according to a de- 
gree of surrounding of said block to output signals 
from said blocks and calculates a movement 
amount of the image at that moment in a real-time 40 
manner. 

7. An apparatus according to claim 6, wherein said 
movement vector calculating means extracts the 
blocks which output the movement vectors due to 45 
the vibration as a target for correction from said 
optical flow and executes a convolution mask ar- 
ithmetic operation to output signals from said 
blocks, thereby applying weights to said signals. 

50 

8. An apparatus according to claim 7, wherein the 
convolution mask arithmetic operation is execut- 
ed in a manner such that in the case where the 
blocks which are adjacent to the target blocks in 

the vertical and lateral directions exist, a point 1 55 
is given to the target blocks, and in the case 
where the blocks which are adjacent in the obli- 
que directions exist, a point 0.5 is given, thereby 



performing an arithmetic operation such that the 
point is increased as the target blocks are sur- 
rounded by the other blocks, and the weight of 
the movement vector is increased for the block of 
a high point. 

9. An apparatus according to claim 8, wherein said 
movement vector calculating means is construct- 
ed by a microcomputer. 

1 0. An apparatus according to claim 1 , further having 
correcting means for correcting a vibration on the 
basis of an output of the movement vector calcu- 
lating means. 

11. An apparatus according to claim 10, wherein said 
correcting means variably changes a read-out 
position when image information is read out from 
the image memory on the basis of an output of 
said movement vector calculating means, there- 
by setting off the vibration. 

1 2. A movement vector detecting apparatus compris- 
ing: 

a) movement vector detecting means for div- 
iding an imput image into a plurality of blocks 
and for detecting movement vectors every 
block; and 

b) movement vector calculating means for ex- 
tracting areas in which the same motion is 
performed from the movement vectors from 
said movement vector detecting means and 
for calculating movement vector values in said 
areas by performing a weight averaging proc- 
ess according to position information which 
occupies in the input image of each of said 
blocks. 

1 3. An apparatus according to claim 1 2, wherein said 
movement vector detecting means generates a 
distribution (optical flow) of the movement vec- 
tors on a picture plane and extracts the blocks 
which output the movement vectors due to the vi- 
bration serving as a target for correction from 
said optical flow and performs a weighting proc- 
ess according to a degree of surrounding of said 
blocks to output signals from said blocks, thereby 
calculating a movement amount of the image at 
that moment in a real-time manner. 

14. An apparatus according to claim 1 3, wherein said 
movement vector calculating means extracts the 
blocks which output the movement vectors due to 
the vibration serving as a target for correction 
from said optical flow and executes a convolution 
mask arithmetic operation to output signals from 
said blocks, thereby applying a weight 
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15. An apparatus according to claim 12, further hav- 
ing correcting means for correcting a vibration on 
the basts of an output of said movement vector 
calculating means. 

16. An apparatus according to claim 15, wherein said 
correcting means includes an image memory and 
variably changes a read-out position when image 
information is read out from said image memory 
on the basis of an output of said movement vector 
calculating means, thereby setting off the vibra- 
tion. 

1 7. A movement vector detecting apparatus compris- 
ing: 

a) movement vector detecting means for div- 
iding an input image into a plurality of blocks 
and for detecting movement vectors every 
block; 

b) movement vector calculating means for ex- 
tracting effective areas in which the same mo- 
tion is performed from the movement vectors 
from said movement vector detecting means 
and for calculating movement vector values in 
said areas by executing a weight averaging 
process according to position information 
which occupies in said areas of each of said 
blocks; and 

c) evaluating means for evaluating the move- 
ment vectors in said effective areas calculat- 
ed by said movement vector calculating 
means on the basis of the total number of 
weight coefficients due to said weight aver- 
age. 

18. An apparatus according to claim 17, wherein said 
movement vector detecting means generates a 
distribution (optical flow) of the movement vec- 
tors on a picture plane and extracts the blocks 
which output the movement vectors due to a vi- 
bration serving as a target for correction from 
said optical flow and executes a convolution 
mask arithmetic operation to output signals from 
said blocks, thereby applying a weight. 

19. An apparatus according to claim 18, wherein said 
convolution mask arithmetic operation is execut- 
ed in a manner such that in the case where the 
blocks adjacent to the target blocks in the vertical 
and lateral directions exist, a point 1 is given to 
said target blocks, and in the case where the 
blocks adjacent in the oblique directions exist, a 
point 0.5 is given, thereby raising the point as the 
numberof other blocks which surround the target 
blocks is large, and the weight of the movement 
vector is increased as the point of the blocks is 
large. 



20. An apparatus according to claim 1 9, wherein said 
evaluating means has a predetermined evalua- 
tion function, and when the total numberof points 
(total number of weight coefficients) is small, 

5 there is a possibility of an erroneous detection, 
so that a small coefficient less than 1 is multi- 
plied, thereby reducing an influence. 

21. An apparatus according to claim 20, wherein as- 
10 suming that the number of blocks in the picture 

plane is set to M x N, said evaluating means ex- 
presses a point distribution by 
0 ^ total number of points ^ 7Mn - 4M - 4N 
+ 2 

15 

22. An apparatus according to claim 17, wherein said 
evaluating means evaluates the reliability of each 
block by calculating a correlation among picture 
planes which differ with respect to the time for a 

20 weight coefficient distribution in a vibrationproof 
area. 

23. A vibrationproof apparatus comprising: 

a) movement vector detecting means for de- 
25 tecting movement vectors every area ob- 
tained by dividing an image into a plurality of 
areas; 

b) vibration amount detecting means for de- 
tecting a vibration amount on the basis of 

30 movement vector information in a designated 

vibrationproof area from said movement vec- 
tor detecting means; 

c) boundary detecting means for detecting a 
boundary of the image; 

35 d) vibrationproof area deciding means for de- 

ciding said vibrationproof area on the basis of 
an output of said boundary detecting means 
and an output of said movement vector detect- 
ing means; and 
40 e) correcting means for correcting a vibration 

in accordance with the information from said 
vibration amount detecting means. 

24. An apparatus according to claim 23, further hav- 
45 ing image pickup means for photoelectrical^ con- 
verting an object image formed on an image pick- 
up surface and for outputting an image pickup 
signal. 

so 25. An apparatus according to claim 24, wherein said 
input image is an image formed on the image 
pickup surface of said image pickup means, and 
said movement vector detecting means detects 
the movement vectors from a luminance signal in 

55 the image pickup signal. 

26. An apparatus according to claim 23, wherein said 
movement vector detecting means generates a 
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distribution (optical flow) of the movement vec- 
tors on a picture plane. 

27. An apparatus according to claim 26, wherein said 
vibrationproof area deciding means discriminates 5 
the area in which a main object to be photograph- 
ed exists by adding said optical flow for a prede- 
termined time. 

28. An apparatus according to claim 27, wherein said w 
boundary detecting means detects a boundary of 

the image on the basis of the result of a Laplacian 
arithmetic operation of the movement vector dis- 
tribution. 

15 

29. An apparatus according to claim 28, wherein said 
boundary detecting means specifies the block in 
which the movement vectors cannot correctly be 
detected from the magnitude of the result of the 
Laplacian arithmetic operation of the movement 20 
vector distribution. 

30. An apparatus according to claim 29, wherein said 
vibrationproof area deciding means decides a 
coarse vibrationproof area from the distribution of 25 
the result in which the optical flow has been add- 
ed for a predetermined time, and further the area 

in which the accurate movement vectors can be 
detected is selected from the result of the Lapla- 
cian arithmetic operation. 30 

31. An apparatus according to claim 30, further hav- 
ing correcting means for correcting a vibration 
on.the basis of an output of said vibration amount 
detecting means. 35 



' obtained by dividing an image into a plurality 
of areas; 

b) vibration amount detecting means for de- 
tecting a vibration amount on the basis of 
movement vector information in a designated 
vibrationproof area from said movement vec- 
tor detecting means; 

c) vibrationproof area deciding means for ap- 
plying different weights to the information 
from the movement vector detecting means, 
for adding the weighted information for a pre- 
determined time, and for deciding said vibra- 
tionproof area on the basis of the addition val- 
ues; and 

d) correcting means for correcting a vibration 
in accordance with information from said vi- 
bration amount detecting means. 

36. An apparatus according to claim 35, further hav- 
ing image pickup means for photoelectrical ly con- 
verting an object image formed on an image pick- 
up surface and.for outputting an image pickup 
signal. 

37. An apparatus according to claim 36, wherein said 
input image is an image formed on the image 
pickup surface of said image pickup means, and 
said movement vector detecting means detects 
the movement vectors from said image pickup 
signal. 

38. An apparatus according to claim 35, wherein said 
movement vector detecting means generates a 
distribution (optical flow) of the movement vec- 
tors on a picture plane. 



32. An apparatus according to claim 31 , wherein said 
correcting means is a variable vertical angle 
prism. 

40 

33. An apparatus according to claim 23, wherein said 
boundary detecting means detects a boundary of 
the image from the result of the Laplacian arith- 
metic operation of the input image. 

45 

34. An apparatus according to claim 33, wherein said 
boundary detecting means detects a zero cross- 
ing point indicative of an edge portion of the im- 
age as a result of the Laplacian arithmetic oper- 
ation of the input image, and said vibrationproof so 
area deciding means sets the block including 

said zero crossing point into a non-vibrationproof 
area and sets the other blocks into the vibration- 
proof areas. 

55 

35. A vibrationproof apparatus comprising: 

a) movement vector detecting means for de- 
tecting movement vectors every area which is 



39. An apparatus according to claim 38, wherein said 
vibrationproof area deciding means decides the 
vibrationproof area on the basis of a movement 
vector distribution as a result in which said optical 
flow has been added for a predetermined time. 

40. An apparatus according to claim 38, further hav- 
ing a plurality of memory means for apply ing:dif- 
ferent weights to the information from said move- 
ment vector detecting means and for accumulat- 
ing the weighted information. 

41. An apparatus according to claim 40, wherein the 
movement vector weight addition information 
which is accumulated in said memory means is 
movement vector information corresponding to 
different photographing states. 

42. An apparatus according to claim 41 , wherein said 
different photographing states include a panning 
operation, an object tracking photographing oper- 
ation, a still photographing operation (camera is 
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fixed), and the like. 

43. An apparatus according to claim 41 , further hav- 
ing comparing and integrating means for compar- 
ing and integrating the different weight addition 5 
values of the movement vectors stored in said 
memory means, for presuming the present pho- 
tographing state, for paying a significance to the 
weight addition value in the memory means cor- 
responding to said photographing state, and for 10 
outputting the weight addition value. 

44. An apparatus according to claim 43, wherein said 
vibration proof area deciding means decides the 
vibrationproof area from the magnitudes of the 15 
movement vectors outputted from said compar- 
ing and integrating means. 

45. A video camera apparatus comprising: 

a) image pickup means for photoelectrically 20 
converting an object image-formed on an im- 
age pickup surface and for outputting an im- 
age pickup signal; 

b) movement vector detecting means for de- 
tecting movement vectors from said image 25 
pickup signal every block obtained by dividing 

said image pickup surface into a plurality of 
blocks; 

c) vibration amount detecting means for de- 
tecting a vibration amount on the basis of 30 
movement vector information in the designat- 
ed vibrationproof area from said movement 
vector detecting means; 

d) memory means for storing the result in 
which different weights have been applied to 35 
the information from the movement vector de- 
tecting means and the weighted information 

has been added for a predetermined time in 
accordance with different weighting charac- 
teristics; 40 

e) vibrationproof area deciding means for de- 
ciding the vibrationproof area on the basis of 
the weight addition values stored in said 
memory means; and 

0 correcting means for correcting a vibration 45 
in accordance with the information from said 
vibration amount detecting means. 

46. An image processing apparatus including a 
movement vector detection means for detecting so 
which parts of an input image are moving; and 
means for processing the moving parts of the in- 
put image to compensate for the movement 
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FIG.2B 



( OPTIAL FLOW ) 
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FIG. 6 
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FIG. 9 
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